McDowell was born in Rossville, Tennessee, near Memphis. His parents, who were farmers, died when McDowell was a youth. He started playing guitar at the age of 14 and played at dances around Rossville. Wanting a change from plowing fields, he moved to Memphis in 1926 where he started to work in the Buck-Eye feed mill where they processed cotton into oil and other products.[4] He also had a number of other jobs and played music for tips. Later in 1928 he moved south into Mississippi to pick cotton.